Computers of the von Neumann type architecture have limited computational speed owing to the communication limitations of the single processor. These limitations can be overcome if a plurality of processors are utilized in the calculation and are operated at least partly in parallel. This alternative architecture, however, generally leads to difficulties associated with programming complexity. Therefore, it is often not a good solution. Recently, an entirely different alternative that does not require programming has shown promise. The networking ability of the neurons in the brain has served as a model for the formation of a highly interconnected set of analog processors, called a "neural network" or "neural net" that can provide computational and reasoning functions without the need of formal programming. The neural nets can learn the correct procedure by experience rather than being preprogrammed for performing the correct procedure. The reader is referred to R. P. Lippmann's article "An Introduction to Computing With Neural Nets" appearing on pages 4-21 of the April 1987 IEEE ASSP MAGAZINE (0740-7467/87/0400-0004/$10.00" 1987 IEEE), incorporated herein by reference, for background concerning neural nets.
Neural nets are composed of a plurality of neuron models, analog processors each exhibiting "axon" output signal response to a plurality of "synapse" input signals. In a type of neural net called a "perceptron", each of these processors calculates the weighted sum of its "synapse" input signals, which are respectively weighted by respective weighting values that may be positive- or negative-valued, and responds non-linearly to the weighted sum to generate the "axon" output response. In the present-day development of the integrated electronic circuitry art, the weighted summation of a large number of terms, each of which has resolution that would require plural-bit digital sampling, can be done appreciably faster and at less cost in integrated circuit die area by processing in the analog regime rather than in the digital regime.
Using capacitors to perform weighted summation in accordance with Coulomb's Law provides neural nets of given size operating at given speed that consume less power than those the analog processors of which use resistors to implement weighted summation in accordance with Ohm's Law. Y. P. Tsividis and D. Anastassion in a letter "Switched-Capacitor Neural Networks" appearing in ELECTRONICS LETTERS, Aug. 27th 1987, Vol. 23, No. 18, pages 958,959 (IEE) described one method of implementing weighted summation in accordance with Coulomb's Law. Their method, a switched capacitor method, is useful in analog sampled-data neural net systems.
U.S. patent applications Ser. Nos. 366,838 and 366,839 filed Jun. 15, 1989 by W. E. Engeler, entitled respectively "NEURAL NET USING CAPACITIVE STRUCTURES CONNECTING INPUT LINES AND DIFFERENTIALLY SENSED OUTPUT LINE PAIRS" and "NEURAL NET USING CAPACITIVE STRUCTURES CONNECTING OUTPUT LINES AND DIFFERENTIALLY DRIVEN INPUT LINE PAIRS", and assigned to General Electric Company describe methods of implementing weighted summation in accordance with Coulomb's Law that do not rely on capacitances being switched and avoid the complexity of the capacitor switching elements and associated control lines.
U.S. patent application Ser. No. 366,838 entitled "NEURAL NET USING CAPACITIVE STRUCTURES CONNECTING INPUT LINES AND DIFFERENTIALLY SENSED OUTPUT LINE PAIRS" describes a type of neural net in which each analog synapse input signal voltage drives a respective input line from a low source impedance. Each input line connects via a respective weighting capacitor to each of a plurality of output lines. The output lines are paired, with the capacitances of each pair of respective weighting capacitors connecting a pair of output lines to one of the input lines summing to a prescribed value. A respective pair of output lines is associated with each axonal output response to be supplied from the neural net, and the differential charge condition on each pair of output lines is sensed to generate a voltage that describes a weighted summation of the synapse input signals supplied to the neural net. A respective operational amplifier connected as a Miller integrator can be used for sensing the differential charge condition on each pair of output lines. Each weighted summation of the synapse input signals is then non-linearly processed in a circuit with sigmoidal transfer function to generate a respective axonal output response. This type of neural net is particularly well-suited for use where all input synapse signals are always of one polarity, since the single-polarity synapse input signals may range over the entire operating supply.
U.S. patent application Ser. No. 366,839 entitled "NEURAL NET USING CAPACITIVE STRUCTURES CONNECTING OUTPUT LINES AND DIFFERENTIALLY DRIVEN INPUT LINE PAIRS" describes a type of neural net in which each analog synapse input signal voltage is applied in push-pull from low source impedances to a respective pair of input lines. Each pair of input lines connect via respective ones of a respective pair of weighting capacitors to each of a plurality of output lines. The capacitances of each pair of respective weighting capacitors connecting a pair of input lines to one of the output lines sum to a prescribed value. Each output line is associated with a respective axonal output response to be supplied from the neural net, and the charge condition on each output line is sensed to generate a voltage that describes a weighted summation of the synapse input signals supplied to the neural net. A respective operational amplifier connected as a Miller integrator can be used for sensing the charge condition on each output line. Each weighted summation of the synapse input signals is then non-linearly processed in a circuit with sigmoidal transfer function to generate a respective axonal output response. This type of neural net is better suited for use where input synapse signals are sometimes positive in polarity and sometimes negative in polarity.
U.S. Pat. No. 5,039,871 issued Aug. 13, 1991 to W. E. Engeler, entitled "CAPACITIVE STRUCTURES FOR WEIGHTED SUMMATION, AS USED IN NEURAL NETS" and assigned to General Electric Company describes preferred constructions of pairs of weighting capacitors for neural net layers, wherein each pair of weighting capacitors has a prescribed differential capacitance value and is formed by selecting each of a set of component capacitive elements to one or the other of the pair of weighting capacitors. U.S. Pat. No. 5,039,870 issued Aug. 13, 1991 to W. E. Engeler, entitled "WEIGHTED SUMMATION CIRCUITS HAVING DIFFERENT-WEIGHT RANKS OF CAPACITIVE STRUCTURES" and assigned to General Electric Company describes how weighting capacitors can be constructed on a bit-sliced or binary-digit-sliced basis. These weighting capacitor construction techniques are applicable to neural nets that utilize digital input signals, as will be presently described, as well as being applicable to neural nets that utilize analog input signals.
The neural nets as thus far described normally utilize analog input signals that may be sampled-data in nature. A paper by J. J. Bloomer, P. A. Frank and W. E. Engeler entitled "A Preprogrammed Artificial Neural Network Architecture in Signal Processing" published in December 1989 by the GE Research & Development Center describes the application of ternary samples as synapse input signals to neural network layers, which ternary samples can be generated responsive to single-bit digital samples.
Analog input signal samples are supplied to a neural net layer parallely in time. There is a class of problems that can usefully employ neural nets for their solution, but for which input signal samples are not available parallely in time. The paper "A Preprogrammed Artificial Neural Network Architecture in Signal Processing" describes the application of serially received analog samples to a plural-stage analog delay line to provide at the outputs of the stages the analog input signal samples supplied to a neural net layer parallelly in time. The paper also describes the application of serially received digital samples to a serial-in/parallel out (SIPO) shift register, the outputs of the shift register stages determining the ternary samples applied in parallel as synapse input signals to neural network layers.
There is a class of problems that can usefully employ neural nets for their solution, but for which synapse input signal samples are plural-bit digital samples. Particularly where there are numerous plural-bit digital synapse input signal samples, difficulties are encountered in getting the plural-bit digital samples into the monolithic integrated circuit in which a neural net layer reposes, because of practical limitations on the number of pins available on the integrated circuit package. An example of when this type of problem is encountered is when one attempts to use a neural net to perform the two-dimensional spatial filtering associated with the recognition of shapes in video signals generated by raster scanning a focal plane array of photosensors in a video camera. In such two-dimensional spatial filtering it is desirable to provide for the temporary storage of the raster-scanned video signal samples, so that they may be temporally aligned for application to the neural net. In the present state of technology the temporary storage of a substantial number of video samples is most advantageously done in digital memory. The synapse input signals made available to the neural net from the digital memory are in digital, not analog, form.
The problem of the synapse input signals made available to a neural net being in plural-bit digital form rather than in analog form can arise in other contexts, too. In systems where the neural net layers are not located in proximity to each other the axon output response of a hidden neural net layer can be digitized in an analog-to-digital converter and transferred without error over substantial distances to another neural net layer. Using digital rather than analog signals to communicate between neural net layers can also be useful when back-propagation training algorithms are used to train a neural net, since it easier in the digital regime to alter the training algorithm in specified ways to take into account whether a new task is being learned or whether instead an old task is being modified to take into account a change in the operating environment.